Content relevance in a social networking system using quality controlled human raters

ABSTRACT

A social networking system builds a quality controlled and desired population-representative pool of human raters to provide ratings on content items to improve a feed ranking model used for providing its users with more relevant content. The system identifies a pool of candidate human raters for providing ratings on a feed of content items. For each candidate human rater of the pool of candidate human raters, the system presents a feed of content items based on a feed ranking model, obtains ratings on the feed of content items, and determines a score representing the consistency of the obtained ratings, the representativeness of the pool of human raters, or the relevance of the content provided by the ranking model. The system uses the computed scores to modify the ranking model used to present content to its users for improving the relevance of the presented content.

BACKGROUND

This disclosure relates generally to improving relevance of contentpresented to a user, such as a social networking system user, and moreparticularly to improving content relevance based on content ratings bya pool of human raters.

Certain online systems, such as social networking systems, allow usersto connect to and to communicate with other users of the system. For asocial networking system, for example, users create profiles on thesocial networking system that are tied to their identities and includeinformation about the users, such as interests and demographicinformation. The social networking system selects and presents contentto a user to encourage the user to interact with the social networkingsystem and with other users of the social networking system. Forexample, the social networking system generates a feed of content itemsfor presentation to a user that includes content items describingactions performed by other users of the social networking system orcontent provided to the social networking system by other users of thesocial networking system.

When selecting content for presentation to a user, social networkingsystems can determine measures of relevance between various contentitems and the user. The measure of relevance between a user and acontent item is typically based at least in part on the user'slikelihood of interacting with the content item when the content item ispresented. The measure of relevance can often be determined usingmachine intelligence by, for example, applying an automated rankingsystem and machine learning model to determine the most relevant contentfor a user. However, in some cases, there can be factors that a humanuser might consider in a relevance determination that are not consideredin a machine-based system.

SUMMARY

An online system, such as a social networking system, builds a qualitycontrolled and representative test population of human raters to providecontent ratings on content items in their news feeds to improve a feedranking model for providing its users with more relevant content intheir news feeds. For example, the human raters are quality controlledby testing the raters against themselves to ensure consistency. Theraters are modified to be representative of a user population ofinterest to the social networking system (e.g., general population of acountry or a user population of the social networking system within thecountry). The raters allow the social networking system to testvariations of the feed ranking model and to run simulations with thosevariations to improve a feed ranking model in determining the relevanceof content presented to its user population of interest.

The social networking system identifies a pool of candidate human ratersfor providing ratings on a feed of content items. For each candidatehuman rater of the pool of candidate human raters, the social networkingsystem presents a feed of content items based on a feed ranking model,obtains ratings on the feed of content items, and determines a scorerepresenting the consistency of the obtained ratings, therepresentativeness of the pool of human raters, or the relevance of thecontent provided by the ranking model.

The social networking system determines a consistency score for eachcandidate human rater that represents a degree of consistency betweenthe various ratings provided by the candidate human rater. Theconsistency score includes two or more components associated withconsistency factors such as person consistency, number-text consistency,temporal consistency, and “bake off” consistency. For example, theconsistency score is determined by computing various component scoresassociated with the consistency factors and the consistency score iscomputed by aggregating the various component scores. The consistencyscores are then used to select a pool of human raters from theidentified pool of candidate human raters such that the ratings of theselected pool of human raters can be used for modifying the feed rankingmodel to improve the relevance of the presented content.

In one embodiment, the social networking system determines arepresentativeness score for the pool of human raters (e.g., the poolselected after performing quality checks) that represents a degree ofsimilarity between the pool of human raters and the user population ofinterest (e.g., a score indicating how representative the human raterpool is of the user population of interest, such as all people in theU.S., all social networking system users in North America, etc.). Therepresentativeness score includes two or more components associated withrepresentativeness factors such as age, gender, amount of time spent onthe social networking system, amount of likes, and a percentage ofcontent associated with friends as opposed to pages. For example, therepresentativeness score is determined by computing various componentscores associated with the representativeness factors and by aggregatingthe various component scores. The representativeness score of theidentified pool is then used to revise the pool of human raters suchthat the revised pool has a sufficient degree of similarity to the userpopulation of interest.

In another embodiment, the social networking system determines arelevance score for the feed ranking model that represents a degree ofrelevance of the content presented via the feed ranking model to thepool of human raters (e.g., pool selected after performing qualitychecks and modified using representativeness score). The relevance scoreincludes two or more components associated with relevance factors suchas person relevance, content relevance, type of impact relevance, amountof impact relevance, a type of content relevance, entertainment valuerelevance, informative value relevance, and an overall relevance. Forexample, the relevance score is determined by computing variouscomponent scores associated with the relevance factors and byaggregating the various component scores.

In a further embodiment, the social networking system tests variationsto the feed ranking model and computes a relevance score associated witheach of the tested variations by analyzing ratings of the pool of humanraters corresponding to content items of each of the tested variations.The social networking system selects one of the feed ranking modelvariations that results in a high degree of relevance to the pool ofhuman raters and in turn uses the selected variation to provide contentitems for its user population of interest, thereby increasing thecontent relevance for its user population of interest.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of a system environment in which a socialnetworking system operates, in accordance with an embodiment.

FIG. 2 is a block diagram of a social networking system, in accordancewith an embodiment.

FIG. 3 is a block diagram of an example feed of content items presentedto a user of a social networking system, in accordance with anembodiment.

FIG. 4 is a block diagram of a content rating system, in accordance withan embodiment.

FIG. 5 is a flowchart of a process for rating content items of thesocial networking system by human raters of the content rating system ofFIG. 4, in accordance with an embodiment.

FIG. 6 is a block diagram of a user interface illustrating a bake offtask for determining consistency of a human rater, in accordance with anembodiment.

FIG. 7 is a block diagram illustrating an interaction between the socialnetworking system of FIG. 2 and the content rating system of FIG. 4, inaccordance with an embodiment.

FIG. 8 is a flowchart of a process for selecting a pool of qualitycontrolled human raters for improving content relevance in a socialnetworking system, in accordance with an embodiment.

FIG. 9 is a flowchart of a process for selecting a pool ofrepresentative human raters for improving content relevance in a socialnetworking system, in accordance with an embodiment.

FIG. 10 is a flowchart of a process for selecting a feed ranking modelfor improving content relevance in a social networking system, inaccordance with an embodiment.

The figures depict various embodiments of the present disclosure forpurposes of illustration only. One skilled in the art will readilyrecognize from the following discussion that alternative embodiments ofthe structures and methods illustrated herein may be employed withoutdeparting from the principles of the disclosure described herein.

DETAILED DESCRIPTION System Architecture

FIG. 1 is a block diagram of a system environment 100 for a socialnetworking system 140. The system environment 100 shown by FIG. 1comprises one or more client devices 110, a network 120, one or morethird-party systems 130, and the social networking system 140. Inalternative configurations, different and/or additional components maybe included in the system environment 100. The embodiments describedherein can be adapted to online systems presenting content to users thatare not social networking systems.

The client devices 110 are one or more computing devices capable ofreceiving user input as well as transmitting and/or receiving data viathe network 120. In one embodiment, a client device 110 is aconventional computer system, such as a desktop or a laptop computer.Alternatively, a client device 110 may be a device having computerfunctionality, such as a personal digital assistant (PDA), a mobiletelephone, a smartphone or another suitable device. A client device 110is configured to communicate via the network 120. In one embodiment, aclient device 110 executes an application allowing a user of the clientdevice 110 to interact with the social networking system 140. Forexample, a client device 110 executes a browser application to enableinteraction between the client device 110 and the social networkingsystem 140 via the network 120. In another embodiment, a client device110 interacts with the social networking system 140 through anapplication programming interface (API) running on a native operatingsystem of the client device 110, such as IOS® or ANDROID™.

The client devices 110 are configured to communicate via the network120, which may comprise any combination of local area and/or wide areanetworks, using both wired and/or wireless communication systems. In oneembodiment, the network 120 uses standard communications technologiesand/or protocols. For example, the network 120 includes communicationlinks using technologies such as Ethernet, 802.11, worldwideinteroperability for microwave access (WiMAX), 3G, 4G, code divisionmultiple access (CDMA), digital subscriber line (DSL), etc. Examples ofnetworking protocols used for communicating via the network 120 includemultiprotocol label switching (MPLS), transmission controlprotocol/Internet protocol (TCP/IP), hypertext transport protocol(HTTP), simple mail transfer protocol (SMTP), and file transfer protocol(FTP). Data exchanged over the network 120 may be represented using anysuitable format, such as hypertext markup language (HTML) or extensiblemarkup language (XML). In some embodiments, all or some of thecommunication links of the network 120 may be encrypted using anysuitable technique or techniques.

One or more third party systems 130 may be coupled to the network 120for communicating with the social networking system 140, which isfurther described below in conjunction with FIG. 2. In one embodiment, athird party system 130 is an application provider communicatinginformation describing applications for execution by a client device 110or communicating data to client devices 110 for use by an applicationexecuting on the client device. In other embodiments, a third partysystem 130 provides content or other information for presentation via aclient device 110. A third party system 130 may also communicateinformation to the social networking system 140, such as advertisements,content, or information about an application provided by the third partysystem 130.

FIG. 2 is a block diagram of an architecture of the social networkingsystem 140. The social networking system 140 shown in FIG. 2 includes auser profile store 205, a content store 210, an action logger 215, anaction log 220, an edge store 225, a content selection module 230, afeed ranking module 235, a quality check module 240, arepresentativeness module 245, a feed relevance module 250, and a webserver 255. In other embodiments, the social networking system 140 mayinclude additional, fewer, or different components for variousapplications. Conventional components such as network interfaces,security functions, load balancers, failover servers, management andnetwork operations consoles, and the like are not shown so as to notobscure the details of the system architecture.

Each user of the social networking system 140 is associated with a userprofile, which is stored in the user profile store 205. A user profileincludes declarative information about the user that was explicitlyshared by the user and may also include profile information inferred bythe social networking system 140. In one embodiment, a user profileincludes multiple data fields, each describing one or more attributes ofthe corresponding social networking system user. Examples of informationstored in a user profile include biographic, demographic, and othertypes of descriptive information, such as work experience, educationalhistory, gender, hobbies or preferences, location and the like. A userprofile may also store other information provided by the user, forexample, images or videos. In certain embodiments, images of users maybe tagged with information identifying the social networking systemusers displayed in an image, with information identifying the images inwhich a user is tagged or stored in the user profile of the user. A userprofile in the user profile store 205 may also maintain references toactions by the corresponding user performed on content items in thecontent store 210 and stored in the action log 220.

While user profiles in the user profile store 205 are frequentlyassociated with individuals, allowing individuals to interact with eachother via the social networking system 140, user profiles may also bestored for entities such as businesses or organizations (e.g., brandpage). This allows an entity to establish a presence on the socialnetworking system 140 for connecting and exchanging content with othersocial networking system users. The entity may post information aboutitself, about its products or provide other information to users of thesocial networking system 140 using a brand page associated with theentity's user profile. Other users of the social networking system 140may connect to the brand page to receive information posted to the brandpage or to receive information from the brand page. A user profileassociated with the brand page may include information about the entityitself, providing users with background or informational data about theentity.

The content store 210 stores objects that each represent various typesof content. Examples of content represented by an object include a pagepost, a status update, a photograph, a video, a link, a shared contentitem, a gaming application achievement, a check-in event at a localbusiness, a brand page, or any other type of content. Social networkingsystem users may create objects stored by the content store 210, such asstatus updates, photos tagged by users to be associated with otherobjects in the social networking system 140, events, groups orapplications. In some embodiments, objects are received from third-partyapplications or third-party applications separate from the socialnetworking system 140. In one embodiment, objects in the content store210 represent single pieces of content, or content “items.” Hence,social networking system users are encouraged to communicate with eachother by posting text and content items of various types of media to thesocial networking system 140 through various communication channels.This increases the amount of interaction of users with each other andincreases the frequency with which users interact within the socialnetworking system 140.

The action logger 215 receives communications about user actionsinternal to and/or external to the social networking system 140,populating the action log 220 with information about user actions.Examples of actions include adding a connection to another user, sendinga message to another user, uploading an image, reading a message fromanother user, viewing content associated with another user, andattending an event posted by another user. In addition, a number ofactions may involve an object and one or more particular users, so theseactions are associated with the particular users as well and stored inthe action log 220.

The action log 220 may be used by the social networking system 140 totrack user actions on the social networking system 140, as well asactions on third party systems 130 that communicate information to thesocial networking system 140. Users may interact with various objects onthe social networking system 140, and information describing theseinteractions is stored in the action log 220. Examples of interactionswith objects include: commenting on posts, sharing links, checking-in tophysical locations via a client device 110, accessing content items, andany other suitable interactions. Additional examples of interactionswith objects on the social networking system 140 that are included inthe action log 220 include: commenting on a photo album, communicatingwith a user, establishing a connection with an object, joining an event,joining a group, creating an event, authorizing an application, using anapplication, expressing a preference for an object (“liking” theobject), and engaging in a transaction. Additionally, the action log 220may record a user's interactions with advertisements on the socialnetworking system 140 as well as with other applications operating onthe social networking system 140. In some embodiments, data from theaction log 220 is used to infer interests or preferences of a user,augmenting the interests included in the user's user profile andallowing a more complete understanding of user preferences.

The action log 220 may also store user actions taken on a third partysystem 130, such as an external website, and communicated to the socialnetworking system 140. For example, an e-commerce website may recognizea user of a social networking system 140 through a social plug-inenabling the e-commerce website to identify the user of the socialnetworking system 140. Because users of the social networking system 140are uniquely identifiable, e-commerce websites, such as in the precedingexample, may communicate information about a user's actions outside ofthe social networking system 140 to the social networking system 140 forassociation with the user. Hence, the action log 220 may recordinformation about actions users perform on a third party system 130,including webpage viewing histories, advertisements that were engaged,purchases made, and other patterns from shopping and buying.Additionally, actions a user performs via an application associated witha third party system 130 and executing on a client device 110 may becommunicated to the action logger 215 by the application for recordationand association with the user in the action log 220.

In one embodiment, the edge store 225 stores information describingconnections between users and other objects on the social networkingsystem 140 as edges. Some edges may be defined by users, allowing usersto specify their relationships with other users. For example, users maygenerate edges with other users that parallel the users' real-liferelationships, such as friends, co-workers, partners, and so forth.Other edges are generated when users interact with objects in the socialnetworking system 140, such as expressing interest in a page on thesocial networking system 140, sharing a link with other users of thesocial networking system 140, and commenting on posts made by otherusers of the social networking system 140.

In one embodiment, an edge may include various features eachrepresenting characteristics of interactions between users, interactionsbetween users and objects, or interactions between objects. For example,features included in an edge describe a rate of interaction between twousers, how recently two users have interacted with each other, a rate oran amount of information retrieved by one user about an object, ornumbers and types of comments posted by a user about an object. Thefeatures may also represent information describing a particular objector user. For example, a feature may represent the level of interest thata user has in a particular topic, the rate at which the user logs intothe social networking system 140, or information describing demographicinformation about the user. Each feature may be associated with a sourceobject or user, a target object or user, and a feature value. A featuremay be specified as an expression based on values describing the sourceobject or user, the target object or user, or interactions between thesource object or user and target object or user; hence, an edge may berepresented as one or more feature expressions.

The edge store 225 also stores information about edges, such as affinityscores for objects, interests, and other users. Affinity scores, or“affinities,” may be computed by the social networking system 140 toapproximate a user's interest in an object or in another user in thesocial networking system 140 based on the actions performed by the user.A user's affinity may be computed by the social networking system 140 toapproximate the user's interest in an object, in a topic, or in anotheruser in the social networking system 140 based on actions performed bythe user. Computation of affinity is further described in U.S. patentapplication Ser. No. 12/978,265, filed on Dec. 23, 2010, U.S. patentapplication Ser. No. 13/690,254, filed on Nov. 30, 2012, U.S. patentapplication Ser. No. 13/689,969, filed on Nov. 30, 2012, and U.S. patentapplication Ser. No. 13/690,088, filed on Nov. 30, 2012, each of whichis hereby incorporated by reference in its entirety. Multipleinteractions between a user and a specific object may be stored as asingle edge in the edge store 225, in one embodiment. Alternatively,each interaction between a user and a specific object is stored as aseparate edge. In some embodiments, connections between users may bestored in the user profile store 205, or the user profile store 205 mayaccess the edge store 225 to determine connections between users.

The content selection module 230 selects one or more content items forcommunication to a client device 110 to be presented to a user. Contentitems eligible for presentation to the user are retrieved from thecontent store 210 or from another source by the content selection module230, which selects one or more of the content items for presentation tothe user. A content item eligible for presentation to the user is acontent item associated with at least a threshold number of targetingcriteria satisfied by characteristics of the user or is a content itemthat is not associated with targeting criteria. Example targetingcriteria include, whether the content item describes a user of thesocial networking system 140 that is connected to the user, whether theuser previously accessed information describing an entity represented inthe social networking system 140 that is described in the content item,whether the user interacted with another content item that is related tothe current content item, and the like. One example of a content item isa newsfeed story that can include different fields or sets ofinformation, such as a name of a user's friend or connection who postedthe story or an object within the social networking system 140 withwhich the friend interacted (e.g., “John Smith invited Bob Roberts toJohn's 21st Birthday Party” or “John Smith liked Big Coffee website”).The story can include one or more graphics, can include links to contenton or off of the social networking system 140, can include icons orfields with which the user can interact (e.g., a Like button, a Sharebutton, a Comments field, etc.), can include sponsored content or anadvertisement, and so forth.

The content selection module 230 invokes a feed ranking model to rankthe content items being presented to the user. The content selectionmodule 230 may present a subset of the content items based on theranking, for example, the top 10 stories, depending on the display areaavailable on a user interface for presenting the content items. Thecontent selection module 230 presents the content items in the orderdetermined by the ranking, for example, content items ranked higher maybe presented more prominently compared to content items ranked lower. Inan embodiment, the content items ranked higher are presented above thecontent items ranked lower. In other embodiments, content items rankedhigher may be presented more prominently by displaying them using anappropriate text color, font, text size, back ground color, etc.

The feed ranking module 235 uses machine learning techniques to generatethe feed ranking model used for ranking the content items. For example,the feed ranking module 235 may generate a ranking model that determinesa ranking score associated with a given content item. The feed rankingmodel can determine a set of content items that are eligible forpresentation to a user (e.g., that meet certain criteria, such as beinga story by a friend or connection of the user), and can order the set ofcontent items based on their ranking scores. The feed ranking modelranks new content items for a user based on information describing theuser's past interactions with other content items (e.g., stories byconnections with which the user frequently interacts on the socialnetworking system 140 may be ranked higher than stories by connectionswith which the user rarely interacts). The information used in theranking process includes data about interactions such as a time spent bythe user on each content item, whether the user performed one or moreactions such as click, like, comments, share, and the like.

The feed ranking module 235 uses the interaction data of the user fromprevious interactions to train a machine learning model to generate aranking of future content items for presenting to the user. The machinelearning model may use supervised learning, where the model is presentedwith a data set of example inputs and their desired outputs such thatmachine learning model can develop a general rule that can map any inputto an output. For example, the machine learning model receives as aninput the past interaction data for the user to develop a general rulefor outputting a likelihood of the user interacting (e.g., like,comment, share etc) with a new content item, which can then be used inranking the content item for presentation to the user. Examples of otherfeed ranking models that the feed ranking module 235 could use to rankcontent items in a newsfeed are described in more detail in U.S. patentapplication Ser. No. 13/194,770 filed Jul. 29, 2011 (issued as U.S. Pat.No. 8,768,863) and U.S. patent application Ser. No. 13/194,773 filedJul. 29, 2011, each of which is incorporated by reference in itsentirety.

The machine learning model develops the general rule by using factorsrelated to interaction rates of a user associated with differentattributes of content items such as an author of the content item and atype of content of the content item (e.g., text, photo, video, etc). Theinteraction rates include time spent on a content item, a click throughrate (CTR) representing a percentage of content items clicked by theuser, like through rate (LTR) representing a percentage of content itemsliked by the user, share through rate (STR) representing a percentage ofcontent items shared by the user with other users, and comment throughrate (CoTR) representing a percentage of content items commented on bythe user. Each attribute of the content item may have a separate set ofinteraction rates for every user. For example, when a new content itemto be ranked is authored by person A and is a photo, a set ofinteraction rates of the user for content items authored by person A andanother set of interaction rates of the user for a photo content typeare used by the machine learning model to determine a set of likelihoodsthat the user would perform interactions with the new content item. Thedetermined set of likelihoods is then used in computing a ranking score,which is used for ranking new content items for presenting to the user.

The ranking score may include various components that correspond to thevarious interaction rates of the content item attributes. Each componenthas a weightage factor that may be different between various components.A ranking model may be varied by varying the weightage factors of theindividual components. An example variation of a ranking model andpresenting of a feed of content items is illustrated below inconjunction with FIG. 3.

The quality check module 240 analyzes ratings provided by candidatehuman raters to perform quality checks for selecting human raters thatmeet a threshold degree of consistency within the provided ratings. Forexample, the quality check module 240 computes a consistency score foreach human rater that represents a degree of consistency for each humanrater across the various ratings provided by the human rater asdescribed below in conjunction with FIG. 7. The social networking system140 selects a pool of human raters from the potential candidates basedon the determined consistency scores as described further below inconjunction with FIG. 8. As described herein, human raters are peoplewho review or interact with content on the social networking system 140and provide ratings on the content according to a set of guidelines asdescribed below in conjunction with FIG. 5. The human raters aretypically users of the social networking system 140 but in some casesneed not be users of the social networking system 140. The content beingrated can be presented to the human raters in the form of a news feed,for example.

The representativeness module 245 analyzes the demographics of and theratings provided by human raters to determine a level of similaritybetween the human raters and the user population of interest the socialnetworking system 140 for selecting a pool of human raters that isrepresentative of a user population of interest. For example, therepresentativeness module 245 determines the level of similarity bycomputing a representative score for the pool of human raters based on aset of representativeness factors as described further below inconjunction with FIG. 7. The social networking system 140 selects andrevises a pool of human raters based on the determinedrepresentativeness score as described further below in conjunction withFIG. 9. The user population of interest to the social networking system140 is a general population of a country in which the social networkingsystem 140 operates, a user population of the social networking system140 within the country, or another population of users of which thesocial networking system would like the raters to be representative intheir makeup as a group.

The feed relevance module 250 analyzes ratings provided by human ratersto determine degrees of relevance of content provided to users usingdifferent feed ranking models for selecting a feed ranking model thatprovides content with sufficient degree of relevance to the userpopulation of interest. For example, the feed relevance module 250determines the degree of relevance by computing relevance scores foreach human rater based on the various ratings provided by the humanrater for content items delivered using the different feed rankingmodels as described further below in conjunction with FIG. 7. The socialnetworking system 140 selects an appropriate or a highest quality feedranking model based on the determined relevance scores as describedfurther below in conjunction with FIG. 10.

The web server 255 links the social networking system 140 via thenetwork 120 to the one or more client devices 110, as well as to the oneor more third party systems 130. The web server 255 serves web pages, aswell as other content, such as JAVA®, FLASH®, XML and so forth. The webserver 255 may receive and route messages between the social networkingsystem 140 and the client device 110, for example, instant messages,queued messages (e.g., email), text messages, short message service(SMS) messages, or messages sent using any other suitable messagingtechnique. A user may send a request to the web server 255 to uploadinformation (e.g., images or videos) that are stored in the contentstore 210. Additionally, the web server 255 may provide applicationprogramming interface (API) functionality to send data directly tonative client device operating systems, such as IOS®, ANDROID™, WEBOS®or BlackberryOS.

Example Presentation of a Feed of Content Items

FIG. 3 is a block diagram of an example feed of content items presentedby a social networking system 140 to a user, in accordance with anembodiment. FIG. 3 shows feeds of content items 310X and 310Y presentedusing different feed ranking models, ranking model X and ranking modelY. The feed of content items 310X is ranked using ranking model X andincludes a plurality of content items 320A-320N selected from a set ofcontent items 320 maintained by the social networking system 140. Thefeed of content items 310Y is ranked using ranking model Y and includesthe plurality of content items 320A-320N in an order different from thatof the feed of content items 310X.

Ranking models X and Y are feed ranking models described above inconjunction with feed ranking module 235 of FIG. 2. Each of the rankingmodels X and Y includes a general rule or collection of rules derived bymachine learning techniques for computing a ranking score for eachcontent item of a user. The ranking score computed under a ranking modelincludes various components that correspond to the various interactionrates for the content item attributes as described above in conjunctionwith feed ranking module 235 of FIG. 2. Each component has a weightagefactor that may be different between various components. A ranking modelmay be varied by varying the weightage factors of the individualcomponents of the ranking score.

In some embodiments, the weightage factors for one or more componentsare set to be different between ranking models X and Y such that rankingmodels X and Y result in different ranking scores for the same contentitem. In these embodiments, the ranking models X and Y generatedifferent ranking scores for a set of content items which results indifferent ranking orders for the same set of content items to bepresented to the user. For example, the top four ranked content items ofthe feed 310X presented using ranking model X, 320A, 320B, 320C, and320D, are different from the top four content items 320C, 320B, 320D,and 320F presented in feed 310Y using ranking model Y.

In some embodiments, the social networking system 140 tests variationsof feed ranking models to present more relevant content to its userpopulation of interest by analyzing ratings on content items presentedby each such test variation of the feed ranking model. An exampleprocess of analyzing ratings from a pool of human raters representativeof the user population of interest to select an appropriate feed rankingmodel is described below in conjunction with FIG. 10.

Content Rating System

FIG. 4 is a block diagram of a system environment 400 for a contentrating system 410, in accordance with an embodiment. The systemenvironment 400 shown by FIG. 4 comprises one or more client devices110, the network 120, and the content rating system 410. In alternativeconfigurations, different and/or additional components may be includedin the system environment 400.

The client devices 110 depicted in FIG. 4 are used by one or more humanraters providing ratings on viewed content. The human raters providetheir content ratings to the content rating system 410 using the clientdevices 110 via the network 120. The content rating system 410 can beincorporated within or separate from the social networking system 140.

The content rating system 410 includes a content rating module 412 and acontent rating store 414. The content rating module 412 receives contentratings from the client devices 110. Example content ratings include anumerical rating on a point scale and/or a text-based rating usingcomments. A numerical rating may be a rating between a range of numberssuch as, for example, one through five, with one representing that thecontent is least relevant to the rater and five representing that thecontent is most relevant to the rater. Numerical ratings, as describedherein, are based on a five point scale ranging from one to five withone representing the least possible rating and five representing thehighest possible rating, unless otherwise specified. A text-based ratingincludes text-based ratings that express the rater's feelings about thecontent item in words.

The content rating store 414 stores objects that each represent varioustypes of content ratings received at the content rating module 412. Inone embodiment, the content rating system 410 is part of the socialnetworking system 140. Alternatively or additionally, the content ratingsystem 410 is a stand-alone system outside of the social networkingsystem 140. The human raters provide ratings on the content theyinteract using an example rating process described below in conjunctionwith FIG. 5.

FIG. 5 is a flowchart of a process for rating content items of thesocial networking system 140 by human raters of the content ratingsystem 410, in accordance with an embodiment. The example process 500may be implemented by various components of the content rating system410. The example process 500 is described with reference to one humanrater and may be repeated for each human rater of the pool of humanraters, where the pool of human raters is identified to represent a userpopulation of interest as described in conjunction with FIGS. 2 and 9.

The social networking system 140 provides each human rater with a feedof content items just as the social networking system 140 would provideeach of its users with a feed of content items. The feed of contentitems includes a batch of content items. For example, each batch ofcontent items includes thirty items. Each human rater provides ratingson a specific number of batches of content items in a given time period(e.g., a day). In the example process 500 of FIG. 5, each human raterprovides ratings for two batches of content items, where each batchincludes thirty items. The human raters provide ratings in accordancewith certain guidelines provided by the social networking system 140.

The client device 110 corresponding to a human rater receives 505 abatch of content items. For each item of the batch of content items, thehuman rater views and/or interacts 510 with the content item as if thehuman rater would interact as a user of the social networking system140. For example, the human rater may like, comment, share, click, ortake any other action while interacting with the content item as thehuman rater would have done if this content item were posted in thehuman rater's news feed of the social networking system 140. The humanrater views and may also interact with the content item before providinga rating for the content item. The human rater provides one or moreratings related to a person involved in the content item, content of thecontent item, an impact of the content item, a content type of thecontent item, entertainment value of the content item, informative valueof the content item, and an overall rating of the content item.

The human rater provides 515 one or more ratings corresponding to aperson and to content associated with the content item. The personrating is a numerical rating representing how much the human rater caresabout the person associated with the content. For example, the numberone represents that the person is someone the human rater really doesnot care about and five represents someone the human rater really caresabout with the other numbers of the number scale representing gradationsbetween the two extremes. The person may be an individual that is theauthor of the content item or an individual that was tagged in thecontent item. For example, the content item is a picture and the personinvolved with the picture is an individual that was tagged in thepicture, which was either posted by the same individual or a differentindividual (irrespective of whether the human rater knows the individualor not).

In one embodiment, the content item may correspond to a brand page onthe social networking system 140 rather than a person. A brand page isfor businesses, brands, and organizations for sharing content associatedwith their organization with users on the social networking system 140.In some embodiments, there may be more than one person involved in thecontent item. Alternatively or additionally, the content item mayfeature one person and a page.

The content rating is also a numerical rating representing how much thehuman rater cares about the content of content item. For example, thenumber one represents that the content is something the human rater isreally not interested in and five represents something the human rateris really interested in, with the other numbers of the number scalerepresenting gradations between the two extremes.

The human rater provides 520 one or more ratings corresponding to animpact the content item has on the human rater. The impact ratings aretypically derived from the person and content rating described above inconjunction with step 515. The human rater provides two ratingsregarding the impact of the content item: type of impact and amount ofimpact. The type of impact rating represents a rating indicating how thehuman rater anticipates using the content item in the human rater'sinteractions on the social networking system 140. For example, the typeof impact rating may include one or more of: a) using the informationfrom the content item in discussions with other people, either online oroffline; b) increasing information about one or more of person, page,product, event, or some combination thereof; c) having a desiredemotional reaction to the content item (e.g., positive or negative); andd) not applicable indicating that there was either no impact or anundesired emotional reaction on the human rater.

The amount of impact rating reflects how much of an impact the contentitem had on the human rater and is related to the type of impact rating.For example, the amount of impact rating is a numerical rating with onerepresenting that the content really doesn't have an impact (or has anegative impact) on the human rater and five representing that thecontent has a significant impact on the human rater, with the othernumbers representing gradations between the two extremes. An examplerelationship between the amount of impact rating and the type of impactrating is described below. For instance if a human rater had a desiredemotional reaction but it was small or at a low level, the amount ofimpact rating is expected to be a rating of three (e.g., a chuckle) asopposed to a rating of five (e.g., laughed hard for a few minutes).

The human rater provides 525 one or more ratings corresponding to anoverall rating of the content item representing how much the human raterwants to see this content item on their news feed. For example, theoverall rating is a numerical rating ranging between the numbers onethrough five, with one representing that the human rater definitely doesnot want to see the content item and five representing that that thehuman rater definitely wants to see the content item, with the othernumbers representing gradations between the two extremes.

The ratings also include one or more comments corresponding to theoverall rating of the content item that relate to one or more attributesof the content item such as the person(s)/page involved, contentinvolved, and type and amount of impact. The comments are text-basedinputs that include reasons why the human rater chose a particularnumerical rating for the person, content, or impact ratings describedabove. For example, the human rater might provide reasons why the personis important (or unimportant) or why the topic of the content item wassomething the human rater really did (or did not) want to know about.The human rater may provide many other reasons for why the human raterwould (or would not) want to see a particular content item in their newsfeed. The provided comments typically correlate with the numericalratings for person(s)/page involved, content involved, and type andamount of impact. The social networking system 140 may perform qualitychecks on the human raters by comparing the numerical rating of aparticular category of rating with its corresponding text-based ratingas described below in conjunction with FIGS. 6 and 7.

In some embodiments, the human rater provides 530 one or more ratingscorresponding to a type of the content involved in the content itemrepresenting whether the human rater categorizes the content as aboutfamily/friends, public content, or both. For example, if a friend sharessomething that is public content, the rating marks the content item as“public.” However, if a friend shares something that is public contentbut the friend also adds additional personal information in the contentitem that makes the overall story seem personal, the human rater marksthe content item as “both.”

In some embodiments, the human rater provides one or more ratingscorresponding to additional questions regarding the human rater'sinteraction with the content item. The additional questions may relateto one or more of how much does the content item: a) entertain the humanrater; b) help the human rater feel connected; c) inform the human raterabout the world; and d) matter to the human rater's life. The humanrater provides rating for each of these questions as a numerical scorewith one representing the least amount of interaction between thecontent item and the human rater and five representing the most amountof interaction, with the other numbers of the scale representinggradations between the two extremes. In addition to providing numericalscores, the human rater also provides text-based ratings for each of theadditional questions similar to the comments corresponding toperson(s)/page involved, content involved, and type and amount of impactdescribed above.

The human rater using a client device 110 accesses the content item andprovides ratings related to steps 505 through 530 for each content itemwithin the batch of content items. The content rating system 410determines whether the human rater provided ratings for all contentitems within a batch of content items. Upon a negative determination,the content rating system 410 repeats the steps 505 through 530. Upon apositive determination, the content rating system 410 then checkswhether the human rater provided ratings for the specified number ofbatches of content items for the day. Upon a negative determination, thecontent rating system 410 provides a new batch of content items(received from the social networking system 140) to the human rater toprovide ratings by repeating steps 505 through 530. Upon a positivedetermination that the human rater provided ratings for the specifiednumber of batches of content items, the content rating system 410indicates to the human rater to wait 540 for a period of time beforeperforming next task of reordering content items. For example, theexample process 500 includes two batches of content items.

After waiting for the period of time, the human rater reorders 545content items that are displayed on the human rater's client device 110.The content rating system 410 displays a set of content items from thetwo batches of content items for the human rater to reorder. The humanrater reorders the displayed content items in an order of overallrelevance to the human rater. For example, if the human rater could onlysee one content item from the set of displayed content items, whichcontent item would it be? Such content item is selected first after thereordering. If the human rater could only see two content items, whichcontent items would they be? Those two content items are selected thefirst and second items. In summary, the human rater reorders the storiesin order of how much the human rater would want to see the items, withthe item the human rater wants to see most first. The reordered list ofcontent items provides an indication of the most relevant content itemsfor the human rater and may also be used for quality checks as describedbelow in conjunction with FIG. 7.

The human rater participates in a quality check task such as “bake off”550 between a series of content item pairs to select a content item ineach pair with which the human rater is more interested to interact. Thecontent items of the content item pairs are selected from one of thebatches of content items the human rater had previously rated during thesame day. The bake off task is described in more detail below inconjunction with FIG. 6.

FIG. 6 is a block diagram of a user interface 600 illustrating a bakeoff task for determining consistency of a human rater, in accordancewith an embodiment. The user interface 600 shows two content items 610and 620 side by side such that the human rater selects one of the twocontent items that the human rater is more interested to interact with.The results from the bake off task are stored in the content ratingsystem 410 that can later be used by the social networking system 140 indetermining a quality check metric for the human rater by comparing thebake off results with that of the human rater's previously providedratings for the content items 610 and 620.

The content item 610 displayed on the left side of the bake off task isfrom a person A 612 with a picture 611. The item 610 includes content614 and a comment 613 provided by person A. The item 610 displaysoptions for the human rater to interact with the items such as like 615,comment 616, and share 617. The content item 620 displayed on the rightside of the bake off task is from a person B 622 with a picture 621. Theitem 620 has incudes content 624 and comment 623 similar to item 610.The item 620 displays options 625, 626, and 627 to interact with theitem similar to that of item 610.

The user interface 600 includes buttons left 630 and right 640 to selectone of the two content items 610 and 620. To select a content item, theuser may select one of the left 630 or right 640 buttons, and then savetheir selection by pressing the save selection 650 button. The bake offtask includes selecting content items for a series of content item pairsand the content rating system 410 stores the selections for each humanrater. The bake off task is repeated for each human rater of the pool ofhuman raters. The ratings provided by the pool of human raters arestored in the content rating store 414 and the stored ratings are usedby one or more components of the social networking system 140 asdescribed below in conjunction with FIG. 7.

FIG. 7 is a block diagram illustrating an interaction between the socialnetworking system of FIG. 2 and the content rating system 410 of FIG. 4,in accordance with an embodiment. The block diagram of FIG. 7 includesthe content rating system 410 of FIG. 4 and the social networking system140 of FIG. 2. The social networking system 140 provides 710 a feed ofcontent items for each of the human raters to the content rating system410. The human raters of the content rating system 410 provide ratingsfor the content items of the feed of content items as described above inconjunction with FIG. 5.

The content rating system 410 provides 720 the content ratings to thesocial networking system 140 for further analysis to improve a relevanceand quality of content provided to the users of the social networkingsystem 140 by analyzing the ratings to modify a feed ranking model usedto provide a feed of content items for improving the relevance of thecontent items provided to its users. The social networking system 140may also modify the pool of human raters used to provide ratings suchthat the modified pool is representative of a user population ofinterest and/or provides content ratings at a minimum level ofconsistency.

The quality check module 240 of the social networking system 140analyzes the ratings received from the content rating system 410 toperform quality checks on the human raters that provided those ratings.The quality check module 240 performs quality checks on the ratings todetermine a degree of consistency for each human rater across thevarious ratings provided by the human rater. For example, the qualitycheck module 240 computes a consistency score for each human rater,where the consistency score represents a degree of consistency acrossthe various ratings provided by the human rater. In one embodiment, theconsistency score includes two or more components associated withconsistency factors such as person consistency, number-text consistencyrepresenting a consistency between numerical ratings and text-basedratings, temporal consistency, and bake off consistency. The qualitycheck module 240 may compute a component score for each of theconsistency components.

The quality check module 240 computes the component score for personconsistency by comparing various ratings associated with a person thatare provided by the human rater. For example, the human rater providesratings for 10 different content items involving person A. The qualitycheck module 240 analyzes the 10 ratings involving person A (numericalratings and/or text-based ratings) to determine a level of consistencybetween the different ratings. If all 10 ratings have similar ratings,the person consistency component score is 1.0. If, on the other hand,only nine of the 10 ratings are similar, the person consistencycomponent score is 0.9. Different ratings may be deemed as similar ifthe ratings convey more or less the same sentiment of the human ratertowards person A. For example, numerical ratings three and four on ascale of five is deemed to be similar. In the same example, numericalratings three and one or three and five are deemed not to be similar andresults in reducing the person consistency component score.

The quality check module 240 computes the component score fornumber-text consistency by comparing a correlation between numericalratings and their corresponding text-based ratings for each contentitem. In one example, the human rater provides a text-based rating and anumerical rating for the content involved in the content item for fivedifferent content items. The quality check module 240 analyzes theratings to determine a level of consistency between the numericalratings and the text-based ratings for each set of the five sets ofratings. A number-text consistency component score is computed based ona percentage of sets of ratings that are deemed to be match. Forexample, if three out of five sets of ratings match, the number-textconsistency component score is 0.6, and if four out of five match, thenumber-text consistency component score is 0.8. A numerical rating isdeemed to match a text-based rating if both the ratings convey more orless the same sentiment of the human rater towards content involved inthe content item. For example, a numerical rating of three on a fivepoint scale for the content is deemed to match a text-based rating thatconveys that the human rater is somewhat interested in the content ofthe content item. In the same example, a numerical rating of four (i.e.,human rater is interested in the content) is deemed not to be match atext-based rating that conveys that the human rater is not interested inthe content and thereby reduces the number-text consistency componentscore.

The quality check module 240 computes the component score for temporalconsistency by checking a consistency of the various ratings provided bythe human rater over a period of time. For example, the human raterprovides ratings for 20 different content items involving person B overa period of time. The quality check module 240 analyzes the 20 ratingsinvolving person B (numerical ratings or text-based ratings) todetermine a level of consistency between the different ratings overtime. If all 20 ratings of the person B are similar, the temporalconsistency component score is 1.0. If, on the other hand, only 10 ofthe 20 ratings are similar, the temporal consistency component score is0.5. Different ratings over time may be deemed as similar if the ratingsconvey more or less the same sentiment of the human rater towards personB. For example, when average numerical rating associated with person Bprovided on day1 is 4.0 and an average numerical rating associated withperson B provided on day2 is 4.2, the temporal consistency componentscore is computed as 95% or 0.95 as the two average ratings are within5% of each other (delta between the average ratings is 0.2, which is 5%of rating 4.0). When the average numerical ratings are 2.0 and 1.5,temporal consistency component score is computed as 75% or 0.75 as thetwo average ratings are 25% apart from each other.

The quality check module 240 computes the component score for bake offconsistency by analyzing results from a bake off task where the humanrater selects a content item that the human rater is more interested tointeract with from a pair of content items. The content items of thecontent item pairs used for the bake off are selected from one of thebatches of content items the human rater had previously rated during thesame day as described above in conjunction with FIG. 6. For each bakeoff task corresponding to a content item pair, the quality check module240 analyzes the ratings provided by the human rater while rating thebatches of content items and also the selection from the bake off task.For example, for the content items 610 and 620 displayed on the userinterface 600 of FIG. 6, the quality check module 240 obtains the savedselection of the preferred item and compares with an overall rating ofthe items 610 and 620. If item 610 has a more preferable overall ratingthan that of item 620 and if the human rater selected item 610 in thebake off task, then the comparison results in a consistent result. Ifthe human rater selected item 620 instead, then the comparison resultsin an inconsistent result.

The quality check module 240 repeats such comparison between the savedselection of the preferred item in the bake off task with that of theoverall ratings for each content item pair involved in the bake off taskto determine a bake off consistency component score. In an example whenthe bake off task includes 10 content item pairs, the quality checkmodule performs 10 such comparisons and the component score isdetermined based on a percentage of consistent results. If all 10comparisons result in consistent score, the bake off consistencycomponent score is 1.0, and if only four of the 10 comparisons result inconsistent score, the bake off consistency component score is 0.4.

The quality check module 240 computes an overall consistency score forthe human rater by aggregating all of the component scores correspondingto the consistency factors such as person consistency, number-textconsistency, temporal consistency, and bake off consistency. In someembodiments, the component scores are multiplied by a weightage factorthat represents how important a particular component is for the overallconsistency score, and the weightage factors are different between thevarious components. For example, the overall human rater consistencyscore may be a summation of the weighted component scores and may becomputed for n components as:

${{consistency}\mspace{14mu} {score}} = {\sum\limits_{k = 1}^{n}\; {{weight}_{k}*{component}_{k}}}$

In the above equation, weight_(k), is the weightage factor for thek^(th) consistency component and component_(k) is the component score ofthe k^(th) consistency component. In the above equation, the ncomponents represent the number of quality checks the quality checkmodule 240 performs for each human rater.

The representativeness module 245 of the social networking system 140analyzes the ratings received from the content rating system 410 andother factors such as the demographics and user profile information todetermine a level of similarity between the pool of human raters of thecontent rating system 410 and a user population of interest to thesocial networking system 140. The representativeness module 245determines the level of similarity by computing a representative scorefor the pool of human raters based on a set of representativenessfactors associated with human raters. An example set ofrepresentativeness factors may include age, gender, amount of time thehuman rater spends on the social networking system 140, an amount oflikes for content generated by the human rater, and a percentage ofcontent in the human rater's timeline associated with the human rater'sfriends as opposed to pages. A human rater's (or any user of the socialnetworking system 140) timeline is a space on the human rater's profileon the social networking system 140 where one can see the human rater'sown content, content from the human rater's friends, and content wherethe human rater is tagged in. The representativeness module 245 maycompute a component score for each of the factors, and then combine theindividual component scores to compute an overall representativenessscore for the pool of human raters.

The representativeness module 245 computes the component score for eachof the factors by comparing a distribution corresponding to each factorbetween the pool of human raters and the user population of interest. Inone embodiment, the distribution of a factor is compared by comparingone or more statistical quantities such as mean. For example, if themean age of the user population of interest is 25 and that of the poolof human raters is 26, the component score for the age factor may be0.96 as the average age of the pool of human raters is different fromthat of the user population of interest by four percent. In someembodiments, other statistical quantities such as standard deviation ora combination of different statistical quantities may be used to computecomponent scores.

The representativeness module 245 computes an overall representativenessscore for the pool of human raters by aggregating all of the componentscores corresponding to each of the representativeness factors. In someembodiments, each component score is multiplied by a weightage factorthat represents how important a particular component is for the overallrepresentativeness score, and the weightage factor is different betweenthe various components. For example, the overall representativenessscore for the pool of human raters may be a summation of the weightedcomponent scores and may be computed for n components as:

${{representativeness}\mspace{14mu} {score}} = {\sum\limits_{k = 1}^{n}\; {{weight}_{k}*{component}_{k}}}$

In the above equation, weight_(k), is the weightage factor for thek^(th) representativeness component and component_(k) is the componentscore of the k^(th) representativeness component. In the above equation,the n components represent the number of representativeness factors usedby the representativeness module 245 for evaluating the similaritybetween the pool of human raters and the user population of interest.

In some embodiments, the representativeness score of the pool of humanraters includes a sub-score for each human rater of the pool of humanraters. The representativeness module 245 computes a representativenesssub-score for each rater of the pool of human raters that indicates alevel of similarity between the human rater and an average user of theuser population of interest. For example, the representativenesssub-score may be computed similar to the representativeness score of thepool of human raters and may include component scores corresponding toeach of the representativeness factors a per human rater level.

The feed relevance module 250 of the social networking system 140analyzes the ratings received from the content rating system 410 todetermine a degree of relevance of content items provided via a feedranking model that provides the feed of content items to the humanrater. The feed relevance module 250 determines the degree of relevanceby computing a relevance score for each human rater based on the variousratings provided by the human rater for content items delivered usingthe feed ranking model. In one embodiment, the relevance score includesone or more relevance components such as person relevance, contentrelevance, type of impact relevance, amount of impact relevance, a typeof content relevance, entertainment value relevance, informative valuerelevance, and an overall relevance. The feed relevance module 250 maycompute a component score for each of the relevance components.

The feed relevance module 250 computes the component score for each ofthe relevance components by determining a numerical value for thereceived rating corresponding to the relevance component. For example,the component score for person relevance is computed by using the personrating provided by the human rater and received from the content ratingsystem 410. The person relevance component score can be computed on anabsolute scale by using the numerical value of the person rating such ason a five point scale. For example, if the person rating is four out offive, then the person relevance component score is also four.Alternatively, the person relevance component score can be computed on arelative scale by using the relative numerical value of the personrating. For the example person rating of four out of five, the personrelevance component score in the relative scale is 0.8 (i.e., fourrelative to five).

The feed relevance module 250 computes an overall relevance score foreach human rater by aggregating all of the component scorescorresponding each of the relevance factors. In some embodiments, eachcomponent score is multiplied by a weightage factor that represents howimportant a particular component is for the overall relevance score, andthe weightage factor is different between the various components. Forexample, the overall relevance score for the human raters may be asummation of the weighted component scores and may be computed for ncomponents as:

${{relevance}\mspace{14mu} {score}} = {\sum\limits_{k = 1}^{n}\; {{weight}_{k}*{component}_{k}}}$

In the above equation, weight_(k), is the weightage factor for thek^(th) relevance component and component_(k) is the component score ofthe k^(th) relevance component. In the above equation, the n componentsrepresent the number of relevance factors used by the feed relevancemodule 250 for determining the relevance of the content items providedby the feed ranking model.

In some embodiments, the feed relevance module 250 computes a relevancescore for the pool of human raters that represents how relevant thecontent items provided by the feed ranking model are to the pool ofhuman raters as a whole. For example, the feed relevance module 250computes the relevance score for the pool of human raters by determiningan average value of the relevance scores for each human rater of thepool of human raters.

Method for Selecting Quality Controlled Human Raters

FIG. 8 is a flowchart of a process 800 for selecting a pool of qualitycontrolled human raters for improving content relevance in a socialnetworking system 140, in accordance with an embodiment. Unlessotherwise noted, the example process 800 of FIG. 8 is performed by thesocial networking system 140, which may store instructions that cause aprocessor on the social networking system 140 to execute the exampleprocess 800. The example process 800 may include additional oralternative steps.

The social networking system 140 receives 805 an indication of a set ofcandidate human raters for providing content ratings. The socialnetworking system 140 may receive the set of candidate human raters fromthe content rating system 410 or from the representativeness module 245of the social networking system 140.

For each candidate human rater of the set of candidate human raters, thesocial networking system 140 presents a feed of content items, receivesratings on the feed of content items, and performs quality checks on thereceived ratings to select a pool of human raters from the set ofcandidate human raters as described in detail below.

The social networking system 140 presents 810 a feed of content items toeach candidate human rater of the set of candidate human raters asdescribed above in conjunction with the content selection module 230 ofFIG. 2. Each candidate human rater provides content ratings on eachcontent item of the feed of content items as described above inconjunction with FIG. 5.

The social networking system 140 receives 815 ratings provided by eachcandidate human rater from the content rating system 410. The receivedratings include ratings on the people involved in the content item, thecontent of the content item, a type and an amount of impact the contentitem has on the candidate human rater, a content type of the contentitem, entertainment value of the content item, informative value of thecontent item, and an overall rating of the content item as describedabove in conjunction with FIG. 5.

The social networking system 140 performs 820 quality checks on thereceived ratings. The quality checks are performed on the receivedratings to determine a degree of consistency for each candidate humanrater across the various ratings provided by the human rater. Examplequality checks include person consistency, number-text consistency,temporal consistency, and bake off consistency as described above inconjunction with FIG. 7.

The social networking system 140 determines 825 a consistency score foreach candidate human rater based on results of the performed qualitychecks on the received ratings. The consistency score includes one ormore components associated with consistency factors such as personconsistency, number-text consistency, temporal consistency, and bake offconsistency, and the consistency score may be computed by aggregatingthe one or more component scores of the consistency factors as describedabove in conjunction with FIG. 7.

The social networking system 140 selects 830 a pool of human raters fromthe set of candidate human raters based on the determined consistencyscores. In one embodiment, the pool of human raters is selected from theplurality of candidate human raters by selecting candidate human raterswith a consistency score above a threshold consistency score. Forexample, the social networking system 140 may determine the thresholdconsistency score as 0.8 or 80%, meaning that a human rater needs to beconsistent at least 80% of the time to pass the threshold consistencyscore. The social networking system 140 may determine the thresholdconsistency score based on machine learning algorithms over time.Alternatively, the social networking system 140 selects the pool ofhuman raters by selecting a percentage of candidate human raters suchthat the selected pool of human raters have a consistency scores higherthan the candidate human raters not selected. For example, if thepercentage is 90%, the top 90% of candidate human raters with higherconsistency scores are selected.

In some embodiments, the candidate human raters are selected so as torepresent a user population of interest to the social networking system140. Additionally or alternatively, the social networking system 140selects the pool of human raters such that the pool of human ratersrepresent the user population of interest to the social networkingsystem 140. For example, the social networking system 140 may deletesome candidate human raters from the pool of human raters that do notrepresent the user population of interest as described in conjunctionwith representativeness module 245 of FIG. 7 and the example process 900of FIG. 9.

Method for Selecting Human Raters Representative of User Population ofInterest

FIG. 9 is a flowchart of a process 900 for selecting a pool ofrepresentative human raters for improving content relevance in thesocial networking system 140, in accordance with an embodiment. Unlessotherwise noted, the example process 900 of FIG. 9 is performed by thesocial networking system 140, which may store instructions that cause aprocessor on the social networking system 140 to execute the exampleprocess 900. The example process 900 may include additional oralternative steps.

The social networking system 140 identifies 905 a pool of human ratersfor providing ratings on a feed of content items of the socialnetworking system 140. The identified pool of human raters represents auser population of interest to the social networking system 140.

For each human rater of the identified pool of human raters, the socialnetworking system 140 presents a feed of content items, obtains ratingson the feed of content items, and determines a representativeness scorefor the pool of human raters in order to revise the identified pool ofhuman raters such that the revised pool better represents the userpopulation of interest as described in detail below.

The social networking system 140 presents 910 a feed of content items toeach human rater of the pool of candidate human raters as describedabove in conjunction with the content selection module 230 of FIG. 2.Each human rater provides content ratings on each content item of thefeed of content items as described above in conjunction with FIG. 5.

The social networking system 140 obtains 915 ratings provided by eachhuman rater of the pool of human raters from the content rating system410. The obtained ratings include ratings on the people involved in thecontent item, the content of the content item, a type and an amount ofimpact the content item has on the candidate human rater, a content typeof the content item, entertainment value of the content item,informative value of the content item, and an overall rating of thecontent item as described above in conjunction with FIG. 5.

In some embodiments, the social networking system 140 performs 920quality checks on the obtained ratings. The quality checks are performedon the obtained ratings to determine a degree of consistency for eachhuman rater across the various ratings provided by the human rater.Example quality checks include person consistency, number-textconsistency, temporal consistency, and bake off consistency as describedabove in conjunction with FIG. 7.

In some embodiments, the social networking system 140 removes 925 one ormore human raters from the pool of human raters based on the qualitychecks performed on the obtained data. For example, the socialnetworking system 140 computes a consistency score for each human raterand selects the human raters to be removed from the pool based on theconsistency score similar to the step 830 described above in conjunctionwith FIG. 8.

The social networking system 140 determines 930 a representativenessscore for the identified pool of human raters based on the obtainedratings data. The representativeness score includes one or morecomponents associated with representativeness factors such as age,gender, amount of time spent on the social networking system 140, amountof likes, and a percentage of content associated with friends as opposedto pages. The representativeness score may be computed by aggregatingthe one or more component scores of the representativeness factors asdescribed above in conjunction with FIG. 7.

The social networking system 140 revises 935 the pool of human ratersbased on the determined representativeness scores. In one embodiment,the revising of the pool of human raters includes either deleting one ormore human raters or adding one or more human raters. The deletingand/or adding of one or more human raters may be based on arepresentativeness sub-score associated with each human rater of thepool of human raters. For example, one or more human raters with arepresentativeness sub-score below a threshold sub-score may be deletedfrom the pool and above the threshold sub-score may be added to thepool. In another example, outlier human raters (e.g., human raters withrepresentativeness sub-scores above and/or below a number of standarddeviations relative to a mean sub-score for the pool of human raters)may be deleted from the pool. A computation of the representativenesssub-score for each human rater is described above in conjunction withrepresentativeness module 245 of FIG. 7.

The social networking system 140 revises 935 the pool of human raters toensure that a representativeness score for the revised pool is above athreshold score. For example, the social networking system 140 maydetermine the threshold representativeness score as 0.9 or 80%, meaningthat the pool of human raters as a group needs to be at least 90%similar with the user population of interest to meet the thresholdscore. The social networking system 140 may determine the thresholdrepresentativeness score based on machine learning algorithms over aperiod of time.

In some embodiments, the social networking system 140 iterates 940 thesteps of presenting a feed of content items, obtaining ratings,determining a representativeness score, and revising the pool of humanraters until the revised pool is similar enough to the user populationof interest. This revision of the pool of raters can be conductedregularly throughout the rating process to ensure that the pool ofraters remains representative of the user population of interest.

Method for Selecting Feed Ranking Model to Improve Feed Relevance

FIG. 10 is a flowchart of a process 1000 for selecting a feed rankingmodel for improving content relevance in a social networking system 140,in accordance with an embodiment. Unless otherwise noted, the exampleprocess 1000 of FIG. 10 is performed by the social networking system140, which may store instructions that cause a processor on the socialnetworking system 140 to execute the example process 1000. The exampleprocess 1000 may include additional or alternative steps.

The social networking system 140 identifies 1005 a pool of human ratersfor providing ratings on a feed of content items of a social networkingsystem 140. The identified pool of human raters represents a userpopulation of interest to the social networking system 140.

For each human rater of the identified pool of human raters, the socialnetworking system 140 presents a first feed of content items based on afeed ranking model, obtains ratings on the first feed of content items,and determines a relevance score for the feed ranking model. The socialnetworking system 140 then repeats the above process by presenting asecond feed of content items based on a modified ranking model,obtaining ratings on the second feed of content items, and determining arelevance score for the modified ranking model. The social networkingsystem 140 selects one of either the feed ranking model or the modifiedfeed ranking model based on the determined relevance scores as describedin detail below.

The social networking system 140 presents 1010 a first feed of contentitems to each human rater of the pool of human raters based on a feedranking model as described above in conjunction with the contentselection module 230 of FIG. 2. Each human rater provides contentratings on each content item of the first feed of content items asdescribed above in conjunction with FIG. 5.

The social networking system 140 receives 1015 ratings provided by eachhuman rater of the pool of human raters for the content item of thefirst feed of content items from the content rating system 410. Thereceived ratings include ratings on the people involved in the contentitem, the content of the content item, a type and an amount of impactthe content item has on the candidate human rater, a content type of thecontent item, entertainment value of the content item, informative valueof the content item, and an overall rating of the content item asdescribed above in conjunction with FIG. 5.

The social networking system 140 determines 1020 a first relevance scorefor each human rater for the feed ranking model based on the receivedratings. The first relevance score includes one or more componentsassociated with relevance factors such as person relevance, contentrelevance, type of impact relevance, amount of impact relevance, a typeof content relevance, entertainment value relevance, informative valuerelevance, and an overall relevance. The first relevance score may becomputed by aggregating the one or more component scores of therelevance factors as described above in conjunction with FIG. 7.

In some embodiments, the social networking system 140 determines thefirst relevance score for the plurality of human raters as a grouprepresenting a degree of relevance of content items provided by the feedranking model. The first relevance score for the group may be determinedby computing an average value of the first relevance scores for eachhuman rater of the plurality of human raters and is referred to as firstgroup relevance score.

The social networking system 140 modifies the feed ranking model to varythe content items presented to the human raters. In one embodiment, thesocial networking system 140 modifies the feed ranking model by varyingan equation that computes a ranking score for each content item to bepresented in a feed of content items. For example, the social networkingsystem 140 varies the equation by changing one or more weightage factorsassociated with components of the ranking score, where each componentcorresponds to various interaction rates of the content item attributessuch as CTR, LTR, CoTR, and STR as describe above in conjunction withFIGS. 2 and 3. The modified feed ranking model may be determined basedon observing the ratings of the raters using the feed ranking model overtime and determining that certain modifications could be made topotentially improve the feed ranking model. For example, if raters tendto be consistently saying that the feeds are skewed toward having toomuch sponsored content and do not have enough stories from connectionsin the social networking system 140, the feed ranking model can bemodified to address these issues.

The social networking system 140 presents 1030 a second feed of contentitems to each human rater of the pool of candidate human raters based onthe modified feed ranking model. Each human rater provides contentratings on each content item of the second feed of content items similarto the ratings provided on the first feed of content items.

The social networking system 140 receives 1035 ratings provided by eachhuman rater of the pool of human raters for the second feed of contentitems from the content rating system 410. The received ratings for thesecond feed are similar to that of the ratings on the first feed andinclude ratings on the people involved in the content item, the contentof the content item, a type and an amount of impact the content item hason the candidate human rater, a content type of the content item,entertainment value of the content item, informative value of thecontent item, and an overall rating of the content item.

The social networking system 140 determines 1040 a second relevancescore for each human rater for the modified feed ranking model based onthe received ratings of the second feed. Similar to the first relevancescore, the second relevance score for the modified ranking model alsoincludes one or more components associated with relevance factors suchas person relevance, content relevance, type of impact relevance, amountof impact relevance, a type of content relevance, entertainment valuerelevance, informative value relevance, and an overall relevance. Thesecond relevance score for the modified feed ranking model may becomputed similar to that of the first relevance score by aggregating theone or more component scores of the relevance factors.

In some embodiments, the social networking system 140 determines thesecond relevance score for the plurality of human raters as a group forthe modified feed ranking model similar to the first group relevancescore of the feed ranking model.

The social networking system 140 selects 1045 one of the feed rankingmodel or the modified feed ranking model for presenting feeds of contentitems based on a comparison between the first relevance score and thesecond relevance score. For example, the social networking system 140selects the ranking model with the highest relevance score, where therelevance score either corresponds to an individual relevance score or agroup relevance score.

In some embodiments, the social networking system 140 iterates 1050 thesteps of modifying the feed ranking model, presenting a feed of contentitems based on the modified ranking model, obtaining ratings,determining a relevance score, and selecting a feed ranking model untilthe selected feed ranking model provides content items with sufficientdegree of relevance to the user population of interest. The socialnetworking system 140 can essentially run a simulation of each modifiedmodel on the raters and compare the ratings of feeds from these modelswith the ratings of feeds from other models. Over time, the socialnetworking system 140 can slowly improve its feed ranking model bymodifying and testing how each modification performs.

Additional Considerations

The foregoing description of the embodiments of the invention has beenpresented for the purpose of illustration; it is not intended to beexhaustive or to limit the invention to the precise forms disclosed.Persons skilled in the relevant art can appreciate that manymodifications and variations are possible in light of the abovedisclosure.

Some portions of this description describe the embodiments of theinvention in terms of algorithms and symbolic representations ofoperations on information. These algorithmic descriptions andrepresentations are commonly used by those skilled in the dataprocessing arts to convey the substance of their work effectively toothers skilled in the art. These operations, while describedfunctionally, computationally, or logically, are understood to beimplemented by computer programs or equivalent electrical circuits,microcode, or the like. Furthermore, it has also proven convenient attimes, to refer to these arrangements of operations as modules, withoutloss of generality. The described operations and their associatedmodules may be embodied in software, firmware, hardware, or anycombinations thereof.

Any of the steps, operations, or processes described herein may beperformed or implemented with one or more hardware or software modules,alone or in combination with other devices. In one embodiment, asoftware module is implemented with a computer program productcomprising a computer-readable medium containing computer program code,which can be executed by a computer processor for performing any or allof the steps, operations, or processes described.

Embodiments of the invention may also relate to an apparatus forperforming the operations herein. This apparatus may be speciallyconstructed for the required purposes, and/or it may comprise ageneral-purpose computing device selectively activated or reconfiguredby a computer program stored in the computer. Such a computer programmay be stored in a non-transitory, tangible computer readable storagemedium, or any type of media suitable for storing electronicinstructions, which may be coupled to a computer system bus.Furthermore, any computing systems referred to in the specification mayinclude a single processor or may be architectures employing multipleprocessor designs for increased computing capability.

Embodiments of the invention may also relate to a product that isproduced by a computing process described herein. Such a product maycomprise information resulting from a computing process, where theinformation is stored on a non-transitory, tangible computer readablestorage medium and may include any embodiment of a computer programproduct or other data combination described herein.

Finally, the language used in the specification has been principallyselected for readability and instructional purposes, and it may not havebeen selected to delineate or circumscribe the inventive subject matter.It is therefore intended that the scope of the invention be limited notby this detailed description, but rather by any claims that issue on anapplication based hereon. Accordingly, the disclosure of the embodimentsof the invention is intended to be illustrative, but not limiting, ofthe scope of the invention, which is set forth in the following claims.

What is claimed is:
 1. A computer-implemented method comprising:receiving an indication of a plurality of candidate human raters forproviding ratings on a feed of content items presented in a socialnetworking system; for each candidate human rater of the plurality ofcandidate human raters: presenting a feed of content items to thecandidate human rater, receiving ratings of the content items providedby the candidate human rater, performing quality checks on the receivedratings of the content items, and determining a consistency score forthe candidate human rater based on the performed quality checks, theconsistency score representing a degree of consistency between thereceived ratings of the candidate human rater, the consistency scoredetermined based on a plurality of consistency factors; and selecting apool of human raters from the plurality of candidate human raters basedon the determined consistency scores.
 2. The computer-implemented methodof claim 1, wherein the received ratings of the content items include atleast one of: numerical ratings on a point scale and text-based ratingsfor a plurality of attributes of the content items.
 3. Thecomputer-implemented method of claim 2, wherein the plurality ofattributes of the content items include at least one of: personinvolved, content, type of impact, amount of impact, a type of content,entertainment value, and informative value.
 4. The computer-implementedmethod of claim 1, wherein the consistency score comprises two or morecomponent scores, each of the two or more component scores associatedwith one of the plurality of consistency factors.
 5. Thecomputer-implemented method of claim 4, wherein the consistency score isa summation of all component scores of the two or more component scores.6. The computer-implemented method of claim 4, wherein the plurality ofconsistency factors comprise at least two of: person consistency,number-text consistency representing a consistency between numericalratings and text-based ratings, temporal consistency representing aconsistency over time, and bake off consistency representing aconsistency between content item pairs.
 7. The computer-implementedmethod of claim 4, wherein each of the two or more component scorescomprise a weightage factor representing a level of importance of acorresponding consistency factor to the overall consistency score. 8.The computer-implemented method of claim 7, wherein a weightage factorcorresponding to a first component score is different from a weightagefactor corresponding to a second component score.
 9. Thecomputer-implemented method of claim 1, wherein selecting the pool ofhuman raters from the plurality of candidate human raters comprisesselecting candidate human raters with a consistency score above athreshold consistency score.
 10. The computer-implemented method ofclaim 1, wherein selecting the pool of human raters from the pluralityof candidate human raters comprises selecting a percentage of candidatehuman raters, the selected candidate human raters having a consistencyscore higher than the candidate human raters not selected.
 11. Thecomputer-implemented method of claim 1, wherein the candidate humanraters represent a user population of interest to the social networkingsystem, the user population of interest representing at least one of: apopulation of a country and a social networking system user populationof a country.
 12. The computer-implemented method of claim 1, whereinselecting the pool of human raters from the plurality of candidate humanraters comprises selecting candidate human raters such that the pool ofhuman raters represent a user population of interest to the socialnetworking system, the user population of interest representing at leastone of: a population of a country and a social networking system userpopulation of a country.
 13. A computer program product comprising anon-transitory computer-readable storage medium having instructionsencoded thereon that, when executed by a processor, cause the processorto: receive an indication of a plurality of candidate human raters forproviding ratings on a feed of content items presented in a socialnetworking system; for each candidate human rater of the plurality ofcandidate human raters: present a feed of content items to the candidatehuman rater, receive ratings of the content items provided by thecandidate human rater, perform quality checks on the received ratings ofthe content items, and determine a consistency score for the candidatehuman rater based on the performed quality checks, the consistency scorerepresenting a degree of consistency between the received ratings of thecandidate human rater, the consistency score determined based on aplurality of consistency factors; and select a pool of human raters fromthe plurality of candidate human raters based on the determinedconsistency scores.
 14. The computer program product of claim 13,wherein the received ratings of the content items include at least oneof: numerical ratings on a point scale and text-based ratings for aplurality of attributes of the content items.
 15. The computer programproduct of claim 14, wherein the plurality of attributes of the contentitems include at least one of: person involved, content, type of impact,amount of impact, a type of content, entertainment value, andinformative value.
 16. The computer program product of claim 13, whereinthe consistency score comprises two or more component scores, each ofthe two or more component scores associated with one of the plurality ofconsistency factors, the plurality of consistency factors comprise atleast two of: person consistency, number-text consistency representing aconsistency between numerical ratings and text-based ratings, temporalconsistency representing a consistency over time, and bake offconsistency representing a consistency between content item pairs. 17.The computer program product of claim 16, wherein the consistency scoreis a summation of all component scores of the two or more componentscores.
 18. The computer program product of claim 16, wherein each ofthe two or more component scores comprise a weightage factorrepresenting a level of importance of a corresponding consistency factorto the overall consistency score, a weightage factor corresponding to afirst component score is different from a weightage factor correspondingto a second component score.
 19. The computer program product of claim13, wherein selecting the pool of human raters from the plurality ofcandidate human raters comprises selecting candidate human raters with aconsistency score above a threshold consistency score.
 20. The computerprogram product of claim 13, wherein selecting the pool of human ratersfrom the plurality of candidate human raters comprises selectingcandidate human raters such that the pool of human raters represent auser population of interest to the social networking system, the userpopulation of interest representing at least one of: a population of acountry and a social networking system user population of a country.